The present invention relates generally to software system performance diagnosis, and more particularly, to correlated tracing with automated multi-layer function instrumentation localization.
In modern large system software, understanding the root cause of system problems is a complicated task. In the execution of application software, multiple software layers such as external libraries and low level system services are involved in addition to the main program binary. The code components of such layers create numerous hidden dependencies which cause the analysis of software problems (e.g., program bugs, performance anomaly) challenging. FIG. 1 depicts software problems and the root cause impact from related components in the same and/or other software layers.
In order to address this challenge, a correlated tracing of multiple software layers (e.g., the application, libraries, and the OS kernel) is necessary for constructing the complete view of program execution paths. By clarifying related events across multiple layers, this information can be a valuable input for application performance debugging to localize root cause problems.
Monitoring application execution has been studied in many publications and patents. Most such approaches meet one of two goals between performance and coverage. Also, these prior approaches are focused on a certain software layer without addressing the monitoring issues systematically across multiple software layers.
For instance, CA's Introscope, http:11www.ca.com/us/application-management.aspx, focuses on a pre-defined set of monitoring functions in the library/API layers (e.g., J2EE/EJB functions). This monitoring scope covers a common set of popularly used functions across the application programs. With this design choice, while this approach can achieve a competent monitoring performance, it does not systematically cover the functions specific to applications or low level system services such as system calls.
The other kind of example is traditional debugging/tracing tools that are popularly used in the debugging and software engineering, such as disclosed by Gdb: the GNU Project debugger, http//sources.redhat.com/gdb, Valgrind: Nicholas Nethercote and Julian Seward. Valgrind: a framework for heavyweight dynamic binary instrumentation. In Proceedings of the 2007 ACM SIGPLAN conference on Programming language design and implementation (PLDI '07). ACM, New York, N.Y., USA, 89-100: and Pin: C.-K. Luk, R. Cohn, R. Muth, H. Patil, A. Klauser, G. Lowney, S. Wallace, V. J. Reddi, and K. Hazelwood. Pin: building customized program analysis tools with dynamic instrumentation. In Proceedings of the 2005 ACM SIGPLAN conference on Programming language design and implementation, PLDI '05, pages 190-200, New York, N.Y., USA, 2005. ACM.
These tools can track software execution in the granularity of instruction. The tracing with this fine-grained granularity provides detailed runtime program information which is very effective for debugging. However, as the downside, these approaches slow down the program execution with an order of magnitude. Therefore they are not suitable for deployed/production systems.
Accordingly, there is a need for request profiling in service systems with kernel events.